System and method for providing a video with lyrics overlay for use in a social messaging environment

ABSTRACT

Some embodiments of the present disclosure provide a server system associated with a media-providing service. The server system receives, from a first client device, video content created by the first client device. The server system receives, from the first client device, an indication that the video content is to be associated with a song provided by the media-providing service. The server system provides, to a second client device, the video content in combination with the song. The server system provides, to the second client device, concurrently with the video content and the song, visual display of metadata about the song, including a name of the song.

CLAIM OF PRIORITY

This application is a continuation of U.S. patent application titled“SYSTEM AND METHOD FOR PROVIDING A VIDEO WITH LYRICS OVERLAY FOR USE INA SOCIAL MESSAGING ENVIRONMENT”, application Ser. No. 16/512,137, filedJul. 15, 2019; which is a continuation of U.S. patent application titled“SYSTEM AND METHOD FOR PROVIDING A VIDEO WITH LYRICS OVERLAY FOR USE INA SOCIAL MESSAGING ENVIRONMENT”, application Ser. No. 15/858,507, filedDec. 29, 2017, issued on Jul. 16, 2019 as U.S. Pat. No. 10,354,633;which claims the benefit of priority to U.S. Provisional patentapplication titled “SYSTEM AND METHOD FOR ASSOCIATION OF A SONG, MUSIC,OR OTHER MEDIA CONTENT WITH A USER'S VIDEO CONTENT”, Application No.62/440,827, filed Dec. 30, 2016; U.S. Provisional patent applicationtitled “SYSTEM AND METHOD FOR PROGRAMMING OF SONG SUGGESTIONS FOR USERSOF A SOCIAL MESSAGING ENVIRONMENT”, Application No. 62/440,833, filedDec. 30, 2016; and U.S. Provisional patent application titled “SYSTEMAND METHOD FOR PROVIDING A VIDEO WITH LYRICS OVERLAY FOR USE IN A SOCIALMESSAGING ENVIRONMENT”, Application No. 62/440,837, filed Dec. 30, 2016;each of which above applications are herein incorporated by reference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF INVENTION

Embodiments of the invention are generally related to providing digitalmedia content, and are particularly related to a system and method forproviding a live lyrics overlay in a social messaging environment.

BACKGROUND

Today's technology-enabled social messaging environments enable users touse lenses or overlays that render over live video, to provide a mode ofself-expression. For example, some social messaging applications supportthe use of video lenses that leverage three-dimensional mappingtechnology, to allow users to render live video of themselves includingan appearance of wearing glasses or dog ears, or of hats or flowers intheir hair, wherein the lenses or overlays map to where the user's headis moving in real time. However, there is currently no mechanism forusers to be able to includes a live lyrics feed in such live videocreations.

SUMMARY

In accordance with an embodiment, described herein is a system andmethod for providing a live lyrics overlay in a social messagingenvironment. The system can utilize advances in three-dimensionalmapping technology that allow social messaging services, to offer realtime video lenses or overlays to their users, and extends thisthree-dimensional mapping technology to support for lyrics. Duringcreation of a video with lyrics lens overlay, the lyrics correspondingto a selected song are retrieved from a lyrics source, and are displayedwithin the video. For example, with the lyrics lens, a user can recordan image of themselves on live video, singing along to a song clip, withthe lyrics of the song displayed as if they appear to be coming fromtheir mouths. The created live lyrics content can also be shared withother users of a social messaging environment.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates an example digital media content environment, inaccordance with an embodiment.

FIG. 2 illustrates a system which enables access to media content from amessaging environment, in accordance with an embodiment.

FIG. 3 illustrates the use of live overlays in a social messagingenvironment, in accordance with an embodiment.

FIG. 4 further illustrates the use of live overlays in a socialmessaging environment, in accordance with an embodiment.

FIG. 5 illustrates the providing of a live lyrics overlay in a socialmessaging environment, in accordance with an embodiment.

FIG. 6 illustrates a process of providing a live lyrics overlay in asocial messaging environment, in accordance with an embodiment.

FIG. 7 illustrates association of a song, music, or other media contentwith a user's video content, to create a video moment, in accordancewith an embodiment.

FIG. 8 further illustrates the association of a song, music, or othermedia content with a user's video content, to create a video moment, inaccordance with an embodiment.

FIG. 9 further illustrates the association of a song, music, or othermedia content with a user's video content, to create a video moment, inaccordance with an embodiment.

FIG. 10 illustrates a process of associating a song, music, or othermedia content with a user's video content, to create a video moment, inaccordance with an embodiment.

FIG. 11 illustrates the programming of song suggestions for users of asocial messaging environment, in accordance with an embodiment.

FIG. 12 further illustrates the programming of song suggestions forusers of a social messaging environment, in accordance with anembodiment.

FIG. 13 further illustrates the programming of song suggestions forusers of a social messaging environment, in accordance with anembodiment.

FIG. 14 illustrates a process of programming song suggestions for usersof a social messaging environment, in accordance with an embodiment.

DETAILED DESCRIPTION

As described above, although some social messaging applications supportthe use of video lenses that leverage three-dimensional mappingtechnology, there is currently no mechanism for users to be able toincludes a live lyrics feed in such live video creations.

In accordance with an embodiment, described herein is a system andmethod for providing a live lyrics overlay in a social messagingenvironment. The system can utilize advances in three-dimensionalmapping technology that allow social messaging services, to offer realtime video lenses or overlays to their users, and extends thisthree-dimensional mapping technology to support for lyrics. Duringcreation of a video with lyrics lens overlay, the lyrics correspondingto a selected song are retrieved from a lyrics source, and are displayedwithin the video. For example, with the lyrics lens, a user can recordan image of themselves on live video, singing along to a song clip, withthe lyrics of the song displayed as if they appear to be coming fromtheir mouths. The created live lyrics content can also be shared withother users of a social messaging environment.

Digital Media Content Environments

FIG. 1 illustrates an example digital media content environment, inaccordance with an embodiment.

As illustrated in FIG. 1 , in accordance with an embodiment, a mediadevice 102, operating as a client device, can receive and play mediacontent provided by a media server system 142 (media server), or byanother system or peer device. In accordance with an embodiment, themedia device can be, for example, a personal computer system, handheldentertainment device, tablet device, smartphone, television, audiospeaker, in-car entertainment system, or other type of electronic ormedia device that is adapted or able to prepare a media content forpresentation, control the presentation of media content, and/or play orotherwise present media content.

In accordance with an embodiment, each of the media device and the mediaserver can include, respectively, one or more physical device orcomputer hardware resources 104, 144, such as one or more processors(CPU), physical memory, network components, or other types of hardwareresources.

Although, for purposes of illustration, a single client media device andmedia server are shown, in accordance with an embodiment a media servercan support the simultaneous use of a plurality of client media devices.Similarly, in accordance with an embodiment, a client media device canaccess media content provided by a plurality of media servers, or switchbetween different media streams produced by one or more media servers.

In accordance with an embodiment, the media device can optionallyinclude a touch-enabled or other type of display screen having a userinterface 106, which is adapted to display media options, for example asan array of media tiles, thumbnails, or other format, and to determine auser interaction or input. Selecting a particular media option, forexample a particular media tile or thumbnail, can be used as a commandby a user and/or the media device, to the media server, to download,stream or otherwise access a corresponding particular media content itemor stream of media content.

In accordance with an embodiment, the media device can also include asoftware media application 108, together with an in-memory client-sidemedia content buffer 110, and a client-side data buffering logic orsoftware component 112, which can be provided as software or programcode that is executable by a computer system or other processing device,and which can be used to control the playback of media content receivedfrom the media server, for playing either at a requesting media device(i.e., controlling device) or at a controlled media device (i.e.,controlled device), in the manner of a remote control.

In accordance with an embodiment, a connected media environment logic orsoftware component 120, which can be provided as software or programcode that is executable by a computer system or other processing device,can be provided at the media device, either as part of the mediaapplication, or separately, for example as a firmware, to enable themedia device to participate within a connected media environment (e.g.,a Spotify Connect environment) that enables a user to control theplayback of media content at such controlled devices.

In accordance with an embodiment, the client-side data buffering logic,together with the media content buffer, enables a portion of mediacontent items, or samples thereof, to be pre-buffered at a client mediadevice. For example, while media options are being prepared for displayon a user interface, e.g., as media tiles or thumbnails, their relatedmedia content can be pre-buffered at the same time, and cached by one ormore media devices in their media content buffers, for prompt andefficient playback when required.

In accordance with an embodiment, the media server can include anoperating system 146 or other processing environment which supportsexecution of a media server 150 that can be used, for example, to streammusic, video, or other forms of media content to a client media device,or to a controlled device.

In accordance with an embodiment, the media server can provide asubscription-based media streaming service, for which a client mediadevice or user can have an associated account and credentials, and whichenable the user's media device to communicate with and receive contentfrom the media server. A received media-access request from a clientmedia device can include information such as, for example, a networkaddress, which identifies a destination media device to which the mediaserver should stream or otherwise provide media content, in response toprocessing the media-access request.

For example, a user may own several media devices, such as a smartphoneand an audio speaker, which can play media content received from a mediaserver. In accordance with an embodiment, identifying informationprovided with a media-access request can include an identifier, such asan IP address, MAC address, or device name, which identifies that themedia-access request is intended for use with a particular destinationdevice. This allows a user, for example, to use their smartphone as acontrolling device, and their audio speaker as a controlled device towhich media content should be sent. The media server can then send therequested media and/or forward the media-access request to the audiospeaker, even though the request originated at the user's smartphone.

In accordance with an embodiment, one or more application interface(s)148 can receive requests from client media devices, or from othersystems, to retrieve media content from the media server. A contextdatabase 162 can store data associated with the presentation of mediacontent by a client media device, including, for example, a currentposition within a media stream that is being presented by the mediadevice, or a playlist associated with the media stream, or one or morepreviously-indicated user playback preferences. The media server cantransmit context information associated with a media stream to a mediadevice that is presenting that stream, so that the context informationcan be used by the device, and/or displayed to the user. The contextdatabase can be used to store a media device's current media state atthe media server, and synchronize that state between devices, in acloud-like manner. Alternatively, media state can be shared in apeer-to-peer manner, wherein each device is aware of its own currentmedia state which is then synchronized with other devices as needed.

For example, in accordance with an embodiment, when the destinationmedia device to which the media content is being streamed changes, sayfrom a controlling device to a controlled device, or from a firstcontrolled device to a second controlled device, then the media servercan transmit context information associated with an active media contentto the newly-appointed destination device, for use by that device inplaying the media content.

In accordance with an embodiment, a media content database 164 caninclude media content, for example music, songs, videos, movies, orother media content, together with metadata describing that mediacontent. The metadata can be used to enable users and client mediadevices to search within repositories of media content, to locateparticular media content items.

In accordance with an embodiment, a server-side media content bufferinglogic or software component 180, which can be provided as software orprogram code that is executable by a computer system or other processingdevice, can be used to retrieve or otherwise access media content items,in response to requests from client media devices or other systems, andto populate a server-side media content buffer 181, at a media deliverycomponent or streaming service 152, which can be similarly provided assoftware or program code that is executable by a computer system orother processing device, with streams 182, 184, 186 of correspondingmedia content data, which can then be returned to the requesting deviceor to a controlled device.

In accordance with an embodiment, a plurality of client media devices,media server systems, and/or controlled devices, can communicate withone another using a network, for example the Internet 190, a local areanetwork, peer-to-peer connection, wireless or cellular network, or otherform of network. For example, a user 192 can interact 194 with the userinterface at a client media device, and issue requests to access mediacontent, for example the playing of a selected music or video item attheir device, or at a controlled device, or the streaming of a mediachannel or video stream to their device, or to a controlled device.

In accordance with an embodiment, the user's selection of a particularmedia option can be communicated 196 to the media server, via theserver's application interface. The media server can populate itsserver-side media content buffer at the server 204, with correspondingmedia content, 206 including one or more streams of media content data,and can then communicate 208 the selected media content to the user'smedia device, or to a controlled device as appropriate, where it can bebuffered in a client-side media content buffer for playing at thedevice.

In accordance with an embodiment, and as further described below, thesystem can include a server-side media gateway or access point 220,which can be provided as software or program code that is executable bya computer system or other processing device, or other process orcomponent, which operates as a load balancer in providing access to oneor more servers, for use in processing requests at those servers. Thesystem can enable communication between a client media device and aserver, via an access point at the server, and optionally the use of oneor more routers, to allow requests from the client media device to beprocessed either at that server and/or at other servers.

For example, in a Spotify media content environment, Spotify clientsoperating on media devices can connect to various Spotify back-endprocesses via a Spotify “accesspoint”, which forwards client requests toother servers, such as sending one or more metadataproxy requests to oneof several metadataproxy machines, on behalf of the client or end user.

Access to Media Content from Social Messaging Environments

In accordance with an embodiment, the system can enable access todigital media content via a social messaging environment, for examplethrough the use of a client-side messaging application, or a softwareconversational agent or “bot”. The media server can recommend orotherwise determine items of media content appropriate to a particularuser interaction. The recommended media content can be provided to amedia device, in the form of a link or other type of reference by whichthe user of the device can stream, download, access, or otherwise usethe media content.

FIG. 2 illustrates a system that enables access to media content from asocial messaging environment, in accordance with an embodiment.

As illustrated in FIG. 2 , in accordance with an embodiment, a socialmessaging environment 240 enables, for each user, a user environment 244that provides a user experience, and the use of a messaging application246 that enables interaction by the user with the social messagingenvironment and with other users.

In accordance with various embodiments, different types of socialmessaging environments can be supported, including, for example, socialnetworking or social media environments (e.g., Instagram), which supportthe sharing of picture-based or other messages between different users,or social messaging environments (e.g., Facebook Messenger) whichsupport the use of a media content bot that operates according to a botconversation logic to provide access to a media content environment.

In accordance with an embodiment, the media server can include arecommendation application program interface (API), which is accessible,e.g., via a HTTP URL, and which provides access to a recommendationengine 252 that provides media content recommendations in response tointeractions or conversations with the social messaging environment.

For example, in accordance with an embodiment, during the interaction248 by a user with the social messaging environment, the messagingapplication or bot can access the media server, via the recommendationAPI, and provide a user interaction information 245, to therecommendation engine, for its use in determining a media contentrecommendation 249, and populating 250 the media delivery component orstreaming service with the recommended media content 251.

The media server can then return, as part of the conversation or otherinteraction with the user, a media content recommendation in the form ofa link or other type of reference by which the user of the device canstream, download, access, or otherwise use the media content.

In accordance with an embodiment, the recommendation engine candetermine a media content recommendation based on criteria such as, forexample, a user's prior listening history, or favorite media content;other user characteristics such as demographic information;external-sourced information, such as the current weather at the user'slocations; or other types of criteria.

In accordance with an embodiment, once the link or other type ofreference to access the recommended media content has been returned tothe media device as part of the interaction, it can be processed by themedia device, to request that the recommended media content be returnedeither to the media device or to a controlled device, for playback atthat device.

For example, the link or reference to an item of media content can beprovided as:

-   -   https://open.spotify.com/track/01G6a8O85SXIH05upiDN1N

The above link can then be passed or otherwise processed by a mediaserver web service API, to request the associated media content.

In accordance with various embodiments, the messaging application ormedia content bot can either operate anonymously, or can perform anauthentication or account verification at the media server, for example,to indicate the user as a Spotify user, and customize the responseaccordingly.

Lyrics Lens Overlay

In accordance with an embodiment, the system includes support forproviding a live lyrics overlay in a social messaging environment.

The system can utilize advances in three-dimensional mapping technologythat allow social messaging services, to offer real time video lenses oroverlays to their users, and extends this three-dimensional mappingtechnology to support for lyrics.

During creation of a video with lyrics lens overlay, the lyricscorresponding to a selected song are retrieved from a lyrics source, andare displayed within the video. For example, with the lyrics lens, auser can record an image of themselves on live video, singing along to asong clip, with the lyrics of the song displayed as if they appear to becoming from their mouths. The created live lyrics content can also beshared with other users of a social messaging environment.

FIG. 3 illustrates the use of live overlays in a social messagingenvironment, in accordance with an embodiment.

As illustrated in FIG. 3 , lenses in social messaging applications canleverage advanced three-dimensional (3D) mapping technology, to mapvirtual items to a live video in real time (260). For example, somesocial messaging applications, such as Snapchat, support the use ofvideo lenses that leverage 3D mapping technology, to allow users torender live video of themselves including an appearance of wearingglasses or dog ears, or of hats or flowers in their hair, wherein thelenses or overlays map to where the user's head is moving in real time.

FIG. 4 further illustrates the use of live overlays in a socialmessaging environment, in accordance with an embodiment.

As illustrated in FIG. 4 , in accordance with an embodiment, a messagingapplication (e.g., a Snapchat, or Messenger, etc. application), or auser-generated video content application (e.g., Instagram, etc.),enables the creation of video content, by a user, for example using acamera feature of their media device.

In accordance with an embodiment, the media server can include arecommendation engine, as described above, which can be used to providemedia content recommendations, in response to interactions orconversations with the system or with a social messaging environment.

In accordance with an embodiment, a user can interact with the messagingapplication, or other video creation application, including utilizingits video capabilities, to access their user environment, and create avideo 406.

For example, in accordance with an embodiment, a user of a socialmessaging environment, using a messaging application, can indicate theywish to create a message that includes live lyrics content

In accordance with an embodiment, a lyrics source 410, having a lyricsdatabase 412, can be used to provide lyrics for a selected song 414.Each song can be associated with a song metadata, which allows matchingof an appropriate set of lyrics to that song. The lyrics source can beeither stored at, or provided by, the media server, or by a third-partysource.

In accordance with an embodiment, the song metadata provides anindication, for example a descriptive song title, that a media servercan use to identify a particular song, music, or other media contentthat is stored at, or capable of being provided by, that media server;and to respond accordingly.

For example, in accordance with an embodiment, each particular item ofmedia content can be associated with a 30-second audio clip from arecognizable portion of that item of media content. A media server canprovide access to a 30-second clip API, which receives requests frommedia devices, or other devices or applications, that include aparticular song metadata, and responds by streaming or otherwiseproviding the 30-second audio clip associated with that song.

Similarly, in accordance with an embodiment, the song metadata can beused to identify the appropriate lyrics associated with that particularsong, music, or other media content, or the associated 30-second audioclip.

In accordance with an embodiment, during creation 416 of a video withlyrics lens overlay 420, lyrics corresponding to the selected song areretrieved 422, 424, from the lyrics source, and are displayed asdisplayed lyrics 426, 428 within the video with lyrics lens overlay.

In accordance with an embodiment, a media file, for example a MovingPicture Experts Group (MPEG)-4 (MP4) file, or other type of media fileformat enabled by the capabilities of the media device and which enablesmetadata information to be associated with the video content of thatfile, for example as part of a file header information, can then becreated.

In accordance with an embodiment, once created, the video with lyricslens overlay can be provided, for example as an MP4 file, or other typeof media file or set of data, to one or more users (recipients) 430,where the recipient can either play 432 the video with lyrics lensoverlay, and/or access 434 the media content identified by the songmetadata, at the media server.

FIG. 5 illustrates the providing of a live lyrics overlay in a socialmessaging environment, in accordance with an embodiment.

As illustrated in FIG. 5 , in accordance with an embodiment, a recipientcan play the video with lyrics lens overlay 440, including the displayedlyrics 442, 444.

Embodiments allows the user to better express themselves through themedium of music and lyrics to convey their emotions to friends or lovedones, including, for example, allowing users to map a live lyrics feedto their live video stream in real time as they record themselvessinging along to their favorite songs.

For example, with the lyrics lens, a user can record an image ofthemselves on live video, singing along to a song clip, with the lyricsof the song displayed as if they appear to be coming from their mouths.

In accordance with various embodiments, the techniques described abovecan also be used in combination with features such as the creation ofvideo moments, as described, for example, in U.S. patent applicationtitled “SYSTEM AND METHOD FOR ASSOCIATION OF A SONG, MUSIC, OR OTHERMEDIA CONTENT WITH A USER'S VIDEO CONTENT”; application Ser. No.15/858,301, filed Dec. 29, 2017; or the programming of song suggestions,as described, for example, in U.S. patent application titled “SYSTEM ANDMETHOD FOR PROGRAMMING OF SONG SUGGESTIONS FOR USERS OF A SOCIALMESSAGING ENVIRONMENT”; application Ser. No. 15/858,457, filed Dec. 29,2017; each of which above applications are herein incorporated byreference.

FIG. 6 illustrates a process of providing a live lyrics overlay in asocial messaging environment, in accordance with an embodiment.

As illustrated in FIG. 6 , in accordance with an embodiment, at step450, a media application is provided for use in accessing media contentat a media server, for playing at a media device.

At step 452, a messaging application that enables creation of videocontent (e.g., Snapchat, Messenger, Instagram, or other application orcomponent), and which can be provided as software or program code thatis executable by a computer system or other processing device, isopened, to supplement a video, including a media content (e.g., a song),with displayed lyrics.

At step 454, the media server, or another lyrics resource, is accessedto provide corresponding lyrics associated with the media content.

At step 456, during recording of the video, the selected media content,and the corresponding lyrics associated with the media content as alyrics lens overlay, are incorporated into the video.

At step 458, during playback of the video with lyrics lens overlay, thecorresponding lyrics associated with the media content are displayed inrealtime, within appropriate areas of the displayed video.

In accordance with an embodiment, a method performed in a media devicecan comprise, by means of a media application, obtaining a selected songfrom a media server. The method also comprises, e.g. by means of amessaging or video application and/or using a camera of the mediadevice, creating a live video. The method also comprises, e.g. based onmetadata of the selected song, retrieving lyrics from a lyrics sourcecorresponding to the selected song. The method also comprises,displaying the retrieved lyrics within the video in real-time as alyrics lens overlay using 3D mapping technology. The live video withdisplayed lyrics may be played in a social messaging environment, whererecipient users may view it or play it.

In accordance with an embodiment, a method performed in a media serversystem can comprise, in response to a media content request from a mediadevice, obtaining a selected song from a media server and sending thesong to the media device, e.g. to a media application thereof. Themethod also comprises, e.g. based on metadata of the selected song,retrieving lyrics from a lyrics source corresponding to the selectedsong and sending the lyrics to the media device, e.g. to a messaging orvideo application thereof, for incorporating the lyrics in a video inreal-time as a lyrics lens overlay using 3D mapping technology.

Creation of Video Moments

In accordance with an embodiment, the system includes support forassociation of a song, music, or other media content with a user's videocontent.

The system enables a user to add a song, music, or other media contentthat is associated with an audio clip and a song metadata, of a mediacontent they may be currently listening, or have just listened, to avideo they are about to create, or have created. Using a video capturemode of a messaging application, or other video creation application,the user can associate a song metadata with a video, for example as anoverlay, to create a shareable file or other set of data, generallyreferred to herein as a video moment.

A recipient of the video moment hear the audio clip in combination withthe video content, and can also view the song metadata overlay, todetermine the name of the song and artist that was used in the video, oroptionally access the song at a media server, for further listening bythe recipient.

FIG. 7 illustrates the association of a song, music, or other mediacontent with a user's video content, to create a video moment, inaccordance with an embodiment.

As illustrated in FIG. 7 , in accordance with an embodiment, a socialmessaging environment enables a messaging application, or other videocreation application, to be used by a user, to interact with a userenvironment or with other users of the social messaging environment.

As further illustrated in FIG. 7 , in accordance with an embodiment, themessaging application enables the creation of video content, by a user,for example using a camera feature of their media device.

In accordance with an embodiment, the user can either capture a newvideo content, or can user the functionality of their media device toretrieve a previously-recorded and saved video content.

In accordance with an embodiment, the media server can include arecommendation engine, as described above, which can be used to providemedia content recommendations, in response to interactions orconversations with the system or with a social messaging environment.

In accordance with an embodiment, a user can interact with the socialmessaging environment, and utilize the video capabilities of theirmessaging application, together with a current playing song, last-playedsong, or other song or music 254, as provided by the media server, tocreate 256 a video moment.

In accordance with an embodiment, the video moment 260 can be associatedwith a song metadata 262 indicative of the, e.g., current playing song.

In accordance with an embodiment, the song metadata provides anindication, for example a descriptive song title, that a media servercan use to identify a particular song, music, or other media contentthat is stored at, or capable of being provided by, that media server;and to respond accordingly.

For example, in accordance with an embodiment, each particular item ofmedia content can be associated with a 30-second audio clip from arecognizable portion of that item of media content. A media server canprovide access to a 30-second clip API, which receives requests frommedia devices, or other devices or applications, that include aparticular song metadata, and responds by streaming or otherwiseproviding the 30-second audio clip associated with that song.

In accordance with an embodiment, other information that might be usefulin providing a context for the video moment, for example a locationinformation, or other information provided by sensors with a user'smedia device, can be associated with the video moment.

In accordance with an embodiment, a media file, for example a MovingPicture Experts Group (MPEG)-4 (MP4) file, or other type of media fileformat enabled by the capabilities of the media device and which enablesmetadata information to be associated with the video content of thatfile, for example as part of a file header information, can then becreated.

In accordance with an embodiment, once created, the video moment can beprovided 270, for example as an MP4 file, or other type of media file orset of data, to one or more users (recipients) 280, for example a userparticipating within the social messaging environment, where therecipient can either play 282 the video moment, and/or optionally access284 the media content identified by the song metadata, at the mediaserver.

FIG. 8 further illustrates the association of a song, music, or othermedia content with a user's video content, to create a video moment, inaccordance with an embodiment.

As illustrated in FIG. 8 , in accordance with an embodiment, a user canindicate they wish to create a video moment (290).

A messaging application that enables creation of video content (e.g.,Snapchat, Messenger, Instagram, or other application or component), andwhich can be provided as software or program code that is executable bya computer system or other processing device, is opened, and a song ormusic portion added to a recorded video, to create a video moment (292,294).

For example, a user can long-press the video capture screen of theirmedia device, to add a current playing song to a messaging application,or to a media application or music player (e.g., a Spotify app); or alast-played song if no song is currently playing; to add the songmetadata to the video (296).

FIG. 9 further illustrates the association of a song, music, or othermedia content with a user's video content, to create a video moment, inaccordance with an embodiment.

As illustrated in FIG. 9 , in accordance with an embodiment, once ane.g., long-press action has been completed, a selected song metadata,e.g., song name and/or artist name 300, for the current song, music, orother media content can be added as an overlay to the video, so that therecipient can both hear the song clip, and see the song metadata whenthey view the video.

FIG. 10 illustrates a process of associating a song, music, or othermedia content with a user's video content, to create a video moment, inaccordance with an embodiment.

As illustrated in FIG. 10 , in accordance with an embodiment, at step302, a media application is provided for use in accessing media contentat a media server, for playing at a media device.

At step 304, a messaging application that enables creation of videocontent (e.g., Snapchat, Messenger, Instagram, or other application orcomponent) is opened, and a song or music portion added to a recordedvideo, to create a video moment.

At step 306, a current playing song, last-played song, or other song ormusic, is added as a song metadata to the video moment.

At step 308, upon receipt and playback of the video moment, by a user ofthe social messaging environment, the song metadata can be used inaccessing the media content at the media server.

Programming of Song Suggestions

In accordance with an embodiment, the system includes support forprogramming of song suggestions for users of a social messagingenvironment.

The system enables a user to add media content, for example a song ormusic, to a video, for sharing with other users.

A media server can provide media content recommendations or suggestions,in response to interactions or conversations with the system or with asocial messaging environment. The suggestions can be used to promptusers of social or messaging environments, to create user generatedcontent, thereby increasing the word-of-mouth, or viral distribution andpromotion of such media content.

For example, in accordance with an embodiment, the system can be used toenable a user to add a song, music, or other media content that isassociated with an audio clip and a song metadata, of a media contentthey may be currently listening, or have just listened, to a video theyare about to create, or have created. Using a video capture mode of amessaging application, or other video creation application, the user canassociate a song metadata with a video content. A recipient of the videocan hear the audio clip in combination with the video content, and canalso view the song metadata overlay, to determine the name of the songand artist that was used in the video, or optionally access the song ata media server, for further listening by the recipient.

FIG. 11 illustrates the programming of song suggestions for users of asocial messaging environment, in accordance with an embodiment.

As illustrated in FIG. 11 , in accordance with an embodiment, a socialmessaging environment enables a messaging application, or video creationapplication, to be used by a user, to interact with a user environmentor with other users of the social messaging environment.

As further illustrated in FIG. 11 , in accordance with an embodiment, amessaging application (e.g., a Snapchat, or Messenger, etc.application), or a user-generated video content application (e.g.,Instagram, etc.), enables the creation of video content, by a user, forexample using a camera feature of their media device.

In accordance with an embodiment, the media server can include arecommendation engine, as described above, which can be used to providemedia content recommendations or suggestions, in response tointeractions or conversations with the system or with a social messagingenvironment.

In accordance with an embodiment, a user can interact with the messagingapplication, or other video creation application, including utilizingits video capabilities, to create 312 a video presentation, incombination with a suggested media content 314.

In accordance with an embodiment, the media server can provide 315 a setof recommended or suggested media content options 316, which can bedisplayed on the user interface for viewing and selection by the user,in creating a video presentation 320.

In accordance with an embodiment, the video presentation can beassociated with a song metadata 322 indicative of the selected mediacontent.

In accordance with an embodiment, the song metadata provides anindication, for example a descriptive song title, that a media servercan use to identify a particular song, music, or other media contentthat is stored at, or capable of being provided by, that media server;and to respond accordingly.

For example, in accordance with an embodiment, each particular item ofmedia content can be associated with a 30-second audio clip from arecognizable portion of that item of media content. A media server canprovide access to a 30-second clip API, which receives requests frommedia devices, or other devices or applications, that include aparticular song metadata, and responds by streaming or otherwiseproviding the 30-second audio clip associated with that song.

In accordance with an embodiment, other information that might be usefulin providing a context for the video presentation, for example alocation information, or other information provided by sensors with auser's media device, can be associated with the video presentation.Generally, the actual content of the video is not analyzed for purposesof making a determination of the context associated with that videocontent.

In accordance with an embodiment, a media file, for example a MovingPicture Experts Group (MPEG)-4 (MP4) file, or other type of media fileformat enabled by the capabilities of the media device and which enablesmetadata information to be associated with the video content of thatfile, for example as part of a file header information, can then becreated.

In accordance with an embodiment, once created, the video presentationcan be provided 330, to one or more users (recipients) 340, for exampleas an MP4 file, or other type of media file or set of data, where therecipient can either play 342 the video presentation, and/or optionallyaccess 344 the media content identified by the song metadata, at themedia server.

FIG. 12 further illustrates the programming of song suggestions forusers of a social messaging environment, in accordance with anembodiment.

As illustrated in FIG. 12 , in accordance with an embodiment, a user canindicate they wish to create a video presentation (350). A messagingapplication that enables creation of video content (e.g., Snapchat,Messenger, Instagram, or other application or component), is opened, tocreate a video presentation (352). For example, a user can long-press ona video creation screen to add a music clip to the video that they arecreating.

In accordance with an embodiment, a media server, or other musicstreaming service, can provides a list of suggestions, that can be usedto promote songs and artists, so that they can be distributed further asusers create user generated content (UGC) (354).

During recording of the video presentation, a selection can be made on avideo creation screen, to add a media content (e.g., music or song clip)to the video presentation being created, which enables a metadataassociated with the selected media content to be associated with thevideo presentation (356).

FIG. 13 further illustrates the programming of song suggestions forusers of a social messaging environment, in accordance with anembodiment.

As illustrated in FIG. 13 , during recording of the video presentation,a selection can be made on a video creation screen, to add a mediacontent (e.g., music or song clip) to the video presentation beingcreated, which enables a metadata 360 associated with the selected mediacontent to be associated with the video presentation, so that therecipient 362 can both hear the song clip, and see the song metadatawhen they play 364 the video.

FIG. 14 illustrates a process of programming song suggestions for usersof a social messaging environment, in accordance with an embodiment.

As illustrated in FIG. 14 , in accordance with an embodiment, at step370, a media application is provided for use in accessing media contentat a media server, for playing at a media device.

At step 372, a messaging application that enables creation of videocontent (e.g., Snapchat, Messenger, Instagram, or other application orcomponent), is opened, to create a video presentation.

At step 374, a media server, or other music streaming service, providesa list of suggestions, that can be used to promote songs and artists, sothat they can be distributed further as users create user generatedcontent (UGC).

At step 376, during recording of the video presentation, a selection canbe made on a video creation screen, to add a media content (e.g., musicor song clip) to the video presentation being created, which enables ametadata associated with the selected media content to be associatedwith the video presentation.

At step 378, upon receipt and playback of the video presentation, by auser, the song metadata can be used in accessing the media content atthe media server.

Embodiments of the present invention may be conveniently implementedusing one or more conventional general purpose or specialized digitalcomputer, computing device, machine, or microprocessor, including one ormore processors, memory and/or computer readable storage mediaprogrammed according to the teachings of the present disclosure.Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art.

In some embodiments, the present invention includes a computer programproduct which is a non-transitory storage medium or computer readablemedium (media) having instructions stored thereon/in which can be usedto program a computer to perform any of the processes of the presentinvention. Examples of the storage medium can include, but is notlimited to, any type of disk including floppy disks, optical discs, DVD,CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs,EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards,nanosystems (including molecular memory ICs), or any type of media ordevice suitable for storing instructions and/or data.

The foregoing description of embodiments of the present invention hasbeen provided for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseforms disclosed. Many modifications and variations will be apparent tothe practitioner skilled in the art.

For example, while the techniques described above generally illustrateexamples such as a music streaming service such as Spotify, and streamedmusic or song content, the systems and techniques described herein canbe similarly used with other types of media content environments, andother types of streamed data or media content.

The embodiments were chosen and described in order to best explain theprinciples of the invention and its practical application, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with various modifications that are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. A method, comprising: at a server systemassociated with a media-providing service, the server system having oneor more processors and memory: receiving, from a first client device,video content created by the first client device; receiving, from thefirst client device, an indication that the video content is to beassociated with a song provided by the media-providing service;retrieving, from a lyrics source, lyrics associated with the song;providing for display at the first client device, concurrently with thevideo content created by the first client device, wherein the lyrics aredisplayed at the first client device as a lyrics lens overlay that ismapped to a portion of a user of the first client device and followsmovement of the portion of the user in the video content created by thefirst client device; providing, to a second client device, the videocontent in combination with the song; and providing, to the secondclient device, concurrently with the video content and the song, thelyrics associated with the song as the lyrics lens overlay that ismapped to the portion of the user of the first client device and followsthe movement of the portion of the user in the video content created bythe first client device.
 2. The method of claim 1, further including,prior to receiving the indication from the first client device that thevideo content is to be associated with the song provided by themedia-providing service, providing a list of song suggestions to thefirst client device, wherein the indication is received in response touser selection of the song from the list of song suggestions.
 3. Themethod of claim 1, wherein the video content and the indication arereceived from a messaging application on the first client device andprovided to the second client device via the messaging application onthe second client device.
 4. The method of claim 2, wherein the videocontent is sent from a user of the first client device to a user of thesecond client device.
 5. The method of claim 1, further comprising,providing, to the second client device, concurrently with the videocontent and the song, visual display of metadata about the song,including a name of the song and an artist name.
 6. The method of claim1, wherein providing, to the second client device, the video content incombination with the song includes providing an audio clip of the song.7. The method of claim 1, further comprising, providing, to the secondclient device, access to the song.
 8. A server system associated with amedia-providing service, comprising: one or more processors; and memorystoring instructions for execution by the one or more processors,including instructions for: receiving, from a first client device, videocontent created by the first client device; receiving, from the firstclient device, an indication that the video content is to be associatedwith a song provided by the media-providing service; retrieving, from alyrics source, lyrics associated with the song; providing for display atthe first client device, concurrently with the video content created bythe first client device, the lyrics associated with the song, whereinthe lyrics are displayed at the first client device as a lyrics lensoverlay that is mapped to a portion of a user of the first client deviceand follows movement of the portion of the user in the video contentcreated by the first client device; providing, to a second clientdevice, the video content in combination with the song; and providing,to the second client device, concurrently with the video content and thesong, the lyrics associated with the song as the lyrics lens overlaythat is mapped to the portion of the user of the first client device andfollows the movement of the portion of the user in the video contentcreated by the first client device.
 9. The server system of claim 8,further including, prior to receiving the indication from the firstclient device that the video content is to be associated with the songprovided by the media-providing service, providing a list of songsuggestions to the first client device, wherein the indication isreceived in response to user selection of the song from the list of songsuggestions.
 10. The server system of claim 8, wherein the video contentand the indication are received from a messaging application on thefirst client device and provided to the second client device via themessaging application on the second client device.
 11. The server systemof claim 10, wherein the video content is sent from a user of the firstclient device to a user of the second client device.
 12. The serversystem of claim 8, the instructions further including instructions forproviding, to the second client device, concurrently with the videocontent and the song, visual display of metadata about the song,including a name of the song and an artist name.
 13. The server systemof claim 8, wherein providing, to the second client device, the videocontent in combination with the song includes providing an audio clip ofthe song.
 14. The server system of claim 8, further comprising,providing, to the second client device, access to the song.
 15. Anon-transitory computer-readable storage medium storing instructions forexecution by a server system associated with a media-providing service,the instructions comprising instructions for: receiving, from a firstclient device, video content created by the first client device;receiving, from the first client device, an indication that the videocontent is to be associated with a song provided by the media-providingservice; retrieving, from a lyrics source, lyrics associated with thesong; providing for display at the first client device, concurrentlywith the video content created by the first client device, the lyricsassociated with the song, wherein the lyrics are displayed at the firstclient device as a lyrics lens overlay hat is mapped to a portion of auser of the first client device and follows movement of the portion ofthe user in the video content created by the first client device;providing, to a second client device, the video content in combinationwith the song; and providing, to the second client device, concurrentlywith the video content and the song, the lyrics associated with the songas the lyrics lens overlay that is mapped to the portion of the user ofthe first client device and follows the movement of the portion of theuser in the video content created by the first client device.
 16. Thenon-transitory computer readable storage medium of claim 15, furtherincluding, prior to receiving the indication from the first clientdevice that the video content is to be associated with the song providedby the media-providing service, providing a list of song suggestions tothe first client device, wherein the indication is received in responseto user selection of the song from the list of song suggestions.
 17. Thenon-transitory computer readable storage medium of claim 15, wherein thevideo content and the indication are received from a messagingapplication on the first client device and provided to the second clientdevice via the messaging application on the second client device. 18.The non-transitory computer readable storage medium of claim 15, whereinthe video content is sent from a user of the first client device to auser of the second client device.
 19. The non-transitory computerreadable storage medium of claim 15, the instructions further includinginstructions for providing, to the second client device, concurrentlywith the video content and the song, visual display of metadata aboutthe song, including a name of the song and an artist name.
 20. Thenon-transitory computer readable storage medium of claim 15, whereinproviding, to the second client device, the video content in combinationwith the song includes providing an audio clip of the song.